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Abstract. To use data from a number of different remote sensors in a synergistic 
manner, a multidimensional analysis of the data is necessary. However, prior to this 
analysis, processing to correct for the systematic geometric distorsion characteris- 
tic of each sensor is required. Furthermore, the registration process must be fully 
automated to handle a large volume of data and high data rates. In this paper, 
a conceptual approach towards an operational multisensor registration algorithm 
is presented. The performance requirements of the algorithm are first formulated 
given the spatially, temporally and spectrally varying factors that influence the 
image characteristics and the science requirements of various applications. Sev- 
eral registration techniques that fit within the structure of our algorithm are then 
presented. Their performance was evaluated using a multisensor test data set as- 
sembled from the Landsat TM, SEASAT, SIR-B, TIMS and SPOT sensors. The 
results are discussed and recommendations for future studies are given. 

1. Introduction 

In future years a number of spaceborne remote sensing instruments will be opera- 
tional. These instruments will gather data over a broad range of the electromagnetic 
spectrum allowing scientists to study the physical, chemical and electrical proper- 
ties of the Earth’s environment on a global scale and over an extended period of 
time. To derive geophysical parameters of interest for each of the planned science 
applications, the data collected by these sensors must be combined and analyzed 
in a multidimensional manner. However, the sensors may be on different platforms 
and in different orbits, have different physical characteristics, viewing geometries, 
and data collection and processing systems. Consequently, systematic and nonsys- 
tematic registration errors will exist between coincident multisensor data samples. 
It is a prerequisite for synergistic analysis of these data to remove such errors. 
Furthermore, because of the anticipated large data volume and high data rates of 
future high resolution sensors, it is necessary to develop an automated multisensor 
registration process that requires no or little operator supervision. 
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Considerable experience has already been accumulated in the operational registra- 
tion of Landsat data (e.g. Grebowsky, 1979). However, these techniques are not 
well adapted to the registration of image data from multiple sensors of significantly 
different characteristics operating at different wavelengths. A robust and adaptable 
automated multisensor registration technique must be developed. 

In this paper, a high-level algorithm that integrates several registration techniques 
is presented. First, a formulation of the performance requirements for development 
of an operational algorithm is given. These requirements are derived from the needs 
of several key science applications as well as a review of practical limitations given 
the image characteristics. We then describe the multisensor test data set that has 
been assembled for evaluation of several computational techniques that fit within 
the structure of our algorithm. One registration technique that has been evaluated 
uses high resolution digital elevation models (DEM) of the areas to be registered. 
Others, which operate in the absence of ancillary data, are based on the extraction 
and matching of scene features across the different images to be coregistered. The 
results are discussed and recommendations for future work are given. 

2. Performance Requirements 

2.1 Characterization of the Input Data 

A number of spatially, temporally and spectrally varying factors influence the image 
characteristics and the registration accuracy. 

Due to the finite precision of the estimate of the platform ephemeris and attitude, 
absolute location errors and geometric distortion affect the geometric quality of 
the imagery. Such errors can typically be removed by the use of tiepoints. How- 
ever, nonsystematic errors and tiepointing bias the image location, and a final step 
of precision registration is required to achieve sub-pixel level accuracy. The ge- 
ometric quality of the data is also affected by the presence of topography in the 
observed scene. For an active sensor like a synthetic aperture radar (SAR), pre- 
dominant terrain-induced geometric distortions such as foreshortening and layover 
(Lewis and Me Donald, 1970) constitutes additional difficulties. Rectification of 
these distortions is essential before registration of the data. As an illustration, a 
perspective view of geocoded and rectified multisensor imagery is shown in Figure 
1 using a technique described in (Kwok et al., 1987). 

For sensors on different platforms and in different orbits, the acquired data are 
intially sampled to grids that are more natural to the sensor geometry than that 
of multisensor registration. A common grid for image coregistration, such as an 
Earth-fixed grid, is required. The process of mapping image data into this grid is 
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known as geocoding and has been developed for a variety of sensors including SAR 
(Curlander et ah, 1987). 

Multisensor registration will be affected by the large variability in spatial resolu- 
tion of the data to be registered (from tens of meters (SAR, HIRIS) to kilometers 
(MODIS, HMMR) in the future NASA Earth Observing System (EOS) platform). 
Since resolution defines the ability of a system to discriminate small details within 
a scene, it establishes a limit for the achievable registration accuracy. 

System noise (i.e. thermal noise, quantization noise, bit error noise, etc.) will also 
affect the registration accuracy, because many of the techniques used for registration 
are very sensitive to noise. While all sensors are corrupted by additive noise from 
the receiver electronics, SAR images are additionally corrupted by multiplicative 
noise known as image speckle. Thus the multisensor registration techniques must 
be robust to noise of a variety of statistics. 

An additional consideration in any registration scheme is the scene composition. 
In cases where only a few features can be positively identified across the various 
sensors, the registration accuracy may be seriously inpaired. Furthermore, identifi- 
able features are inherently space, time and frequency dependent. Therefore, it is 
necessary to develop robust automated techniques of selection of invariant features 
across the multisensor data. 

In view of the above remarks, the input and output data requirements for an op- 
erational algorithm can be formulated. They define the operational domain and 
conditions under which the multisensor algorithm is expected to operate, and can 
be used as a basis for the evaluation of candidate algorithms. 

2.2 Input and Output Data Requirements 

The input data shall be corrected from the geometric distortion characteristic of 
each sensor using the best information available, geocoded onto a preselected grid 
common to all sensors (e.g. UTM), and resampled to the same pixel spacing. The 
signal to noise ratio of the data shall be better than 5 dB. The geodetic accuracy 
of the input images shall be better than 500 meters or 10-50 pixels. It is expected 
that most sensors will do better than this since most of them will have an accurate 
geographical location system on board. 

The output products shall have a registration accuracy of less than one resolution 
element. This requirement is derived from a subset of application being considered 
for multisensor data analysis. 
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In the case of change detection, sub-pixel accuracy is desired to compare the re- 
sponse of individual pixel elements. Depending on the scene characteristics (pres- 
ence of identifiable features) this requirement may be very difficult to achieve. In 
other cases, for example in the global study of hydrological cycles (which includes 
tasks such as sea-ice identification and dynamics, determination of moisture content 
of soil and vegetation, vegetation identification, areal extent and growth, etc.) a 
registration accuracy of several resolution elements may be sufficient. 

It is important to point out that although the accuracy requirements have typically 
been well defined for each individual instrument, little or no accuracy requirements 
have yet been clearly defined for multisensor registration by the scientific community 
(EOS, 1987). More work is clearly needed in this area for each interdisciplinary 
science application. 

3. Multisensor Test Data 

A multisensor test data set has been assembled using image products from SEASAT 
SAR, SIR-B SAR, Landsat TM, SPOT, and TIMS (Kwok et al., 1989). Information 
on each sensor, including look angle, spectral range, polarization and spatial reso- 
lution is given in Table 1. Geocoding of the images to a common UTM Earth-grid 
has been performed and the data have been resampled to the same pixel spacing of 
25 meters. Several sub-images of reduced size (512 x 512, 1024 x 1024 pixels) were 
selected from the areas where the sensors have coincident coverage. The characteris- 
tics of the original image data and of the selected sub-images are presented in Table 
2. This table includes information about the geographic location of the data, the 
initial sample spacing and size, the revolution number and date of acquisition, the 
number of selected sub-images and the type of map projection used for coding. A 
summary list of the natural features present in the imaged scenes is also indicated. 

For each selected sub-image, manual registration was performed, resulting in an 
estimated relative misregistration uncertainty of less than ± 2 pixels, roughly equal 
to the largest resolution element (40 meters). This uncertainty results from the 
differences in resolution between the various sensors. This estimate is used as a 
basis for the true registration for quantitative evaluation of the performance of 
various automated registration techniques. 

4. Automated Multisensor Registration 

The structure of the candidate multisensor registration algorithms is presented on 
Table 3. The input data satisfy the requirements as formulated in the previous sec- 
tion. The first processing step consists of automatically selecting sub-frames from 
each input image to define local areas of multisensor coincident coverage where pre- 
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cision registration can be performed with a high confidence of success. Depending 
on the availability of ancillary data (DEM or cartographic maps) a registration 
mode is selected. For the case where DEM is available the multisensor data are 
coregistered to the common grid provided by the DEM. Otherwise, invariant fea- 
tures are extracted from the sub-images and correspondence is established across 
the data to be registered. To reduce the computational complexity of the algorithm 
and obtain several estimates of the misregistration per sub-image, feature match- 
ing is performed at multiple locations and the results are then filtered to evaluate 
their relative spatial consistency within the selected patch (local constraints). If 
the match can be labeled as statistically significant (e.g. satisfies some goodness 
measure), the misregistration error of the selected sub-image is estimated and the 
multisensor data are then registered. Otherwise, the result is rejected and the 
selection and matching process is repeated with different parameters. At a higher- 
level of processing, the combined results from different features and from registered 
neighborhood patches (global constraints) can be used to produce a more accurate 
and more reliable solution. In effect, a cooperative process can be established where 
the results from different stages of the processing are used as reinforcements for the 
entire process. 

Several candidate techniques which are effective within this structure are presented 
in the remainder of this section. They have been selected based on compatibil- 
ity, robustness and adaptivity to the various sensors. Each matching algorithms’ 
performance is assessed using the multisensor test data set described in the last 
section. 

4.1 Automated Selection of Sub-images 

Selection of the patches where fine registration is desired must be based on the 
extraction of stable features that can be unambiguously identified across the entire 
multisensor image data set. The difficulty is to formulate an approach without 
a — priori knowledge of the scene content. One possible technique has been described 
in (Davis and Kenue, 1978) where binary edge maps are used to compute a figure of 
merit for candidate control points. The results obtained with images from different 
sensors are then cross correlated to retain valid candidates. 

4.2. Automated registration to digital terrain data 

Our approach is to simulate multisensor imagery from a digital elevation model 
(DEM) of the area where the sensors have a coincident coverage and register this 
simulated imagery with the actual imagery, thereby inducing coregistration of the 
multisensor data on the common grid provided by the DEM. 
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Using elevation data, viewing geometry and a model of the scene reflectance, the 
appearance of the scene for any given sun angle and viewing angle can be simulated 
as in (e.g. Horn and Bachman, 1978) for passive sensors operating in the visible and 
near-visible part of the spectrum. An example of a synthetic image generated using 
this technique is shown in Figure 2. The illumination parameters were matched to a 
Landsat TM image data acquired over the same area. A simple matching technique 
(area-correlation) is then used to establish the correspondence between the images. 
The registration error is approximately 80 meters for the images shown. 

Our approach to generate simulated SAR image from the DEM is similar to that 
described above. The sensor imaging geometry, the elevation data and a model of 
the radar backscatter are all required to produce the image shown in Figure 3. The 
imaging geometry simulates that of a SEASAT image acquired over the same area. 
An area correlation scheme is then used to match the radar and simulated images. 
Using tiepoint measurements of identifiable features not within the image shown, a 
misregistration error of 60 meters was obtained. 

Several potential error sources affect the registration accuracy, including the the 
uncertainty in the actual imaging geometry, the geometric accuracy of the DEM 
data (height), and the reflectance model used for the optical data. 

4.3 Computational Approaches 

In the absence of reference maps, elevation data, geographical information or cor- 
relative ground truth information, blind techniques based on the identification of 
invariant features across the data can be used for image registration. 

A. Feature extraction 

Candidate features commonly used in digital imagery include edges, regions, lines, 
vertices of line intersection, shapes, etc. These features must be robust to change 
in sensor geometry, wavelength, SNR and noise statistics. Two particular types of 
features, region boundaries and edges, were examined using our multisensor data 

set. 

Multisensor region boundaries extraction 


Region boundaries are one of the simplest invariant low-level features than can be 
used to characterize the misregistration. 

Even though many unsupervised segmentation techniques exist for optical images, 
most of them are not effective for SAR images because of the presence of speckle 
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noise. One unsupervised technique that seems to work reasonably well is a scheme 
based on a clustering algorithm to segment the images into several regions of sim- 
ilar intensity and texture (Kwok et al., 1989). The region boundaries are then 
established where a class transition occurs. 

A resulting segmentation map, using 3 classes, is shown on Figure 4 together with 
the original images from SEASAT, Landsat, and SPOT. A 3 x 3 pixels window 
was used at each pixel location to compute the mean grey level and grey level 
texture via a simple standard deviation measure. The results obtained by matching 
these region boundaries are usually less accurate than those obtained with other 
techniques. However region segmentation can still be refined, especially in the 
case of SAR imagery, to provide information that complements results from other 
techniques. 

Multisensor edge detection 

An extensive literature exists on the subject of edge detection in optical imagery. 
However, in the case of SAR images the detection process is complicated since the 
images are corrupted by speckle noise. Techniques based on an approximation of the 
first and second directional derivatives (e.g. Sobel, or Robert operators) perform 
poorly, especially in terms of localization of the edges since they tend to produce 
large responses. Statistical edge operators (Touzi et al., 1988, Frost et al., 1982) in 
a lot of cases suffer from the same limitation. 

This problem is solved by regularization techniques, specifically using a two-dimensional 
Gaussian smoothing operator as in a Marr-Hildreth operator (Marr and Hildreth 
1980) or a Canny edge detector (Canny 1983). These operators typically have good 
detection and localization properties without multiple responses to a single edge, 
the three performance criteria for evaluation of edge detection algorithms. Theo- 
retically, these techniques are compatible to almost all types of remote sensor data. 
Their performance with optical data have been documented in the literature (Marr 
and Hildreth 1980, Canny 1983). 

The performance of these two operators was quantitatively compared in (Kwok and 
Rignot, 1989) in the case of synthetic SAR images as well as actual SAR images. It 
was shown that the gradient operator outperforms the Laplacian operator in both 
detection and localization of edges in image speckle. 

Significant improvments in the performance of the VG operator can result from 
optimizing the parameter selections. In particular, the value of the filter spatial 
width a must be adapted to the spatial resolution of the different sensors. Automatic 
thresholding is another important factor. In our implementation, a threshold with 
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hysteresis as in (Canny, 1983) is used to eliminate insignificant edges. Further 
post-processing such as thinning and contour-filling techniques have been shown to 
improve the quality of subsequent matches. Another possible improvement of the 
edge detector uses multiple operator widths and combines the resulting edges using 
a technique called feature synthesis, where the responses of the smaller operators 
are used to predict the response of a larger operator. Some results with optical 
images have been presented in (Canny 1983). 

For illustration, one example of edge-map using SEASAT, Landsat TM and SPOT 
data and the Canny edge detector with a spatial width of 2 pixels (40 m) and 
adaptive thresholding is presented in Figure 5. 

B. Feature Matching 

Candidate feature matching techniques include binary cross correlation, distance 
transform / Chamfer matching, dynamic programming, and structural and symbolic 
matching. 

In the case of region boundaries and edges a convenient binary representation of the 
feature maps can be used, a grey level of one at location of a feature-point and zero 
otherwise. This representation reduces the computational complexity of feature 
matching since computational cost becomes proportional to a linear dimension as 
opposed to area correlation where computational cost is proportional to an area. 

Binary correlation 

The binary feature-maps of each of the images to be registered can be cross corre- 
lated for various relative image shifts. The shift corresponding to the peak of the 
cross correlation will be an estimate of the actual misregistration between the im- 
ages. The process is fast and can be efficiently implemented on an array processor 
or vectorizing computer. 

Distance transform and Chamfer matching 

The distance transform and Chamfer matching are described in (Barrow et al. 
1977). In this method feature-points are matched by minimizing a generalized 
distance between them. 

A distance transform is first applied to a binary feature-map, arbitrarily refered 
as the source image. The result of this transformation is a distance map where 
the grey level of each pixel is a measure of the distance between the pixel and the 
nearest feature-point. For various values of the relative shift between the source and 
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the target images, the total distance between the feature points of the two images 
can be computed. This measure is the sum of the distance values read from the 
source image at each location of a feature-point in the target image. If matching 
were perfect, this distance would be zero. The relative shift that produces the 
smallest total distance corresponds to an estimate of the actual relative translational 
misregistration between the images to be registered. 

This method is more robust to distortion or residual rotation effects than a binary 
correlation method. 

Comparison of binary correlation and Chamfer matching 

The time of computation of the binary correlation is less than the time of computa- 
tion of Chamfer matching, typically in the ratio 1 to 4 for a search area of 100 x 100 
pixels using a 512 x 512 pixel image. The tolerance to residual rotation effects is of 
1 degree in the case of the binary cross correlation based on a maximum registration 
accuracy of 2 pixels. This tolerance is improved to 3 degrees when thicker edges are 
used (3 pixels wide instead of 1) (Wong, 1977). In the case of Chamfer matching 
the rotation tolerance is of 3 degrees. Better registration results (10 to 20 %) were 
consistently obtained by binary correlation as compared to Chamfer matching. The 
reason is that the quality metric used during Chamfer matching does not perform as 
well as expected with multisensor data due to the presence of non-matchable edges 
across the data, i.e. edges that appear in one image and not in the other. Their 
presence biases the total distance between feature points and significantly affects 
the accuracy, whereas the binary cross correlation is not affected by non-matchable 
edges. 

Dynamic Programming 

This iterative method, combined with an autoregressive model (AR), was used in 
the work by (Maitre and Wu, 1989) to register severely distorted optical images to 
a reference map without a-priori knowledge of the distortion. The two processes 
work at a different level. The AR model defines the deformation of the image at 
a pixel scale, and dynamic programming optimizes the search for best registration 
of an ordered sequence of features or primitives (usually edges) with a comparable 
sequence of features extracted from a reference map. The technique is robust in 
the presence of non-matchable edges. Good results are shown in (Maitre and Wu, 
1989) using NOAA-7 satellite data. 

Th is method has not been tested yet using the multisensor test data set, but offers 
good potential. 
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C. Constraint Filtering 


In practice, matching is performed on small areas (typically 256 x 256 pixels or less) 
to minimize the distortion. Thus, the time of computation is also reduced and the 
number of estimates of the misregistration between the two images is increased. The 
resulting data must therefore be filtered to eliminate false matches. A clustering 
technique can be used where the cluster centroid corresponds to the estimated 
misregistration of the images. 

At a higher level, results obtained from several feature matches are used to improve 
clustering of the data. Results from neighborhood patches can also be included. 

4.4 Experimental Results 

Twelve 512 x 512 pixel images corresponding to 3 different geographic areas have 
been registered. Each image was divided into 4 sub-blocks, and the search area for 
the local registration shift was 101 x 101 pixels in each sub-block. 

In the case of the images from SEASAT and SPOT, the rate of success of the binary 
correlation of edges was 87 %, and increased to 92 % after constraint filtering, with 
no false matching. In the case of images from SEASAT and Landsat TM, the rate 
of success of the same technique was 85 % before constraint filtering, and 86 % 
after. Registration was qualitatively more difficult in that case because of the lower 
resolution of the Landsat images as compared to SPOT images, and also because a 
few additional scenes where registration was more difficult was used. 

The registration accuracy of the multisensor data was approximately ± 2 pixels (40 
m). The achievability of sub-pixel accuracy seemed difficult to establish by visual 
inspection of our multisensor test data set, a fact that is a common problem when 
comparing digital imagery from multiple remote sensors. 

5. Conclusions and recommendations 

It is of considerable importance to develop automated multisensor registration tools 
for synergistic use of the data from a variety of spaceborne sensors. A high level 
algorithm that integrates a variety of registration techniques in a systematic manner 
was presented in this paper. It was tested using a somewhat limited multisensor 
test data set. A more complete study would enlarge this data set to include more 
instruments and more scene types. Additional techniques for feature extraction and 
feature matching also need to be evaluated in a follow-on study. 

The performance of a multisensor registration algorithm is dependent on the sci- 
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ence requirements of the particular applications as well as the characteristics of the 
instrument, the imaged surface and the environmental conditions. This very com- 
plex task cannot be solved with just a single technique, but will require combining 
several techniques together that work in a competitive-cooperative mode of interac- 
tion. For this reason it seems logical that a rule based artificial intelligence approach 
may be necessary for the high level algorithm to select the optimal techniques and 
parameters from a particular multisensor application. 
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(a) (b) 

Fig. 1. Perspective viewing of multisensor geocoded and rectified images of an area 
near Los Angeles, California: (a) SEAS AT radar image; (b) Landsat TM, band 4, 
image. 
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Fig. 2. Comparison of simulated versus actual Landsat TM image: (a) simulated 
image: (b) actual image. 
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(a) 


(b) 


Fig. 3. Comparison of simulated versus actual SEAS AT radar image: (a) simulated 
image: (b) actual image. 
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Fig. 4. I nsupervised segmentation of Optical and SAR images from an area near 
the Altamaha River, Georgia, (a) SEASAT, (b) Landsat TM, and (c) SPOT images 
are segmented into 3 regions represented in (d), (e), and (f) respectively. The 
corresponding images of region boundaries are (g), (h), and (i) respectively. 
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Fig. 5. Edge detection in (a) SEASAT, (b) Landsat TM, and (c) SPOT images of 
an area near the Altamaha River, Georgia. The edge-maps obtained from Canny’s 
edge detector are represented in (d), (e), and (f) respectively. 



Table 1. Description of the characteristics of the different sensors involved in the 
constitution of the multisensor test data set. 
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MULTISENSOR TEST DATASET 


FEATURES IN 
PATCHES 

Rivers, Lakes, 
Fields, Roads, 
Coasts. 

Mountains, 
Rivers, Lakes, 
Fields, Roads, 
Cities, 

Mountains, 

Rivers, 

Fields, Roads, 
Cities, Dunes. 

Mountains, 

Fields. 

TIMS 




Date: Jul 83 
Size: 1 K x 1 K 
Rotated to North 
# of patches 
selected for 
testing: 4 

SPOT 

Date: Sept 84 
Size: 3 K x 5 K 
Rotated to North 
# of patches 
selected for 
testing: 12 




LANDSAT 

Date: Jul 84 
Size: 3 K x 4 K 
Rotated to North 
# of patches 
selected for 
testing: 12 

Date: Jun 84 
Size: 5 K x 5 K 
Rotated to North 
# of patches 
selected for 
testing: 13 

Date: Jun 84 
Size: 3 K x 4 K 
Rotated to North 
# of patches 
selected for 
testing: 13 

Date: Nov 82 
Size: 3 K x 4 K 
Rotated to North 
# of patches 
selected for 
testing: 4 

SEASAT 

Rev : 407 
Date: Jul 78 
Size: 5 K x 5 K 
Map Proj. : UTM 
# of patches selectee 
for testing: 12 

Rev : 781 
Date: Aug 78 
Size: 5 K x 5 K 
Map Proj. : UTM 
# of patches selected 
for testing: 5 

Rev : 681 
Date: Aug 78 
Size: 3 K x 3 K 
Map Proj. : UTM 
# of patches selected 
for testing: 13 

Rev : 882 
Date: Aug 78 
Size: 5 K x 5 K 
Map Proj. : UTM 
# of patches selected 
for testing: 4 

IMAGE FRAMES 
LOCATION 

ALTAMAHA 

RIVER, 

GEORGIA 

( pixel size = 20 m ) 

WIND RIVER 
BASIN, 
WYOMING 

( pixel size = 30 m ) 

YUMA, 

ARIZONA 

( pixel size = 25 m ) 

DEATH VALLEY, 
CALIFORNIA 

( pixel size = 25 m ) 
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JPL AUTOMATED MULTISENSOR REGISTRATION 
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MISREGISTRATION ESTIMATE 




